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ABSTRACT 

Pursuant to the provisions of the National School 
Lunch Act and Nutrition Amendments of 1977, Tennessee instituted a 
statewide nutrition education program aimed at educators, school food 
service personnel, and children in schools and child care 
institutions. Establishment of an evaluation team early in the 
program development process proved beneficial in that the evaluators 
helped develop measurable program objectives, and were able to assist 
in needs assessment and in the measurement of the base-line 
conditions prior to the program* s implementation. A comparison of the 
nutritional knowledge and nutritional practices of students in 48 
elementary schools revealed that, in a majority cf the grade levels 
evaluated, students who had participated in the program less than a 
year exhibited greater knowledge and better practices than did 
students in schools where the program had not been implemented. 
(Author/PGD) 
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A CHANCE TO DO 'IT RIGHT: ASSESSING THE IMPACT ON 
PARTICIPANTS OF A STATE-WIDE NUTRITION EDUCATION PROGRAM 

Trudy Banta and Jo Lynn Cunningham 
The University of Tennessee, Knoxville 

Program evaluators decry the fact that in entirely too many instances they are - 
invited to enter the scene at a point too late in the life of a project to permit 
them to do their best work. For example, program objectives (if written) may be in 
a form that makes them difficult if not impossible to measure; the program director 
may limit assessment data either directly or indirectly to data s/he chooses to pro- 
vide; and, in the worst possible case, the project may be underway so that the possi- 
bility of gathering baseline data has be'en eliminated. 

The authors of this paper have had the unusual and highly satisfying opportunity 
. during the past two years to build a state-wide evaluation from its inception, work- 
ing with the state program coordinator to cons cruet measurable program objectives based 
on needs identified in a state-wide reeds assessment, specifying an appropriate evalua- 
tion design, working with the team of curriculum specialists who developed objectives 
for students and accompanying instructional materials, designing a series of assess- 
ment instruments, and then using these instruments to test a state-wide sample both 
before and after program implementation. In short, the evaluators had the chance to 
do it right. 

The Tennessee Nutrition Education and Training (NET) Program is a component of 
a national effort to develop a coordinated nutrition education program for children 
from preschool through Grade 12. This effort has received federal funding through 
the U.S. Department of Agriculture. Origins of the program can be traced to Public 
Law 95-166, the National School Lunch Act and Nutrition Amendments of 1977, which 



provided under Section 19 for a program of "Nutrition Education and Training," This 
legislation authorized funding to carry out a nutrition information and education 
program through a system of grants to state agencies to provide for (a) training in 
nutrition for educators and school food service personnel, (b) training in food ser- 
vice management for school food service personnel, and (c) conducting nutrition edu- 
cation activities in schools and child care institutions. 

Early in 1979, the evaluators worked closely with the state coordinator to develop 
broad goals and more specific objectives for the NET program in Tennessee, A needs 
assessment involving public school personnel had been conducted the previous year, 

and the needs identified were addressed in the program plan. Moreover, the NET objec- 

i 

tives were written in terms that made it possible for the evaluators to assess the ex- 
tent to which they had been achieved as the program evolved. 

Evaluation Procedures 

During the 1979-80 academic year, a group of faculty and students at The Univer- 
sity of Tennessee, Knoxville (UTK) spent four months identifying, and then validating 
via consultant review, a set of nutrition education goals and objectives to be attained 
by students in Grades K- 12 in Tennessee, The team included specialists in nutrition 
and food sciences; human development; consumer studies; early childhood, elementary, 
secondary, and adult education; and measurement and evaluation. Early in 1980, the 
same team of specialists constructed developmental ly appropriate instructional mate- 
rials for use by teachers in Grades K-6. 

A series of assessment instruments for students, their parents and teachers, 
school administrators, and food service personnel was devised to correspond to the 
goals and objectives framework. Because previous assessments of achievement associ- 
ated with nutrition education programs have been criticized for dealing only with 
knowledge and neglecting attitudes and practices of program participants, affective 
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and behavioral as well as cognitive components were included in the assessment instru- 
ments for use in Tennessee. Eleven different instruments were constructed: forms for 
students at five developmental levels (Grades K-l, 2-3, 4-6, 7-9, and 10-12), teachers 
at two levels (elementary and secondary), food service personnel in two categories 
(managers and workers), school administrators, and parents. A measure of nutrition 
knowledge was included in nine of the instruments (all but those for administrators 
and food service workers). All instruments contained self-report measures of nutrition 
practices and attitudes as well as perceptions of nutrition education. In addition, 
an observation-based measure of eating behavior (plate waste) was designed for use with 
the five student groups. 

A pretest-posttest control group design was used to assess the impact on partici- 
pants of the implementation of Tennessee's nutrition education curriculum. During 
April 1980, the evaluators sent trained field assistants to elementary and secondary 
schools in each of the state's nine development districts to administer the assessment 
instruments, thus collecting baseline data on nutrition knowledge, attitudes, and 
practices and perceptions of nutrition education. There were 48 schools involved in 
the assessment. In each development district, two elementary schools were designated 
as treatment schools (i.e., teachers, food service personnel, and an administrator 
from each designated to receive training during Summer 1980 in the use of the curricu- 
lum materials); two elementary schools were des d nated as control schools (i.e., no 
training in nutrition education provided for school personnel until Summer 1981). Be- 
cause instructional plans for Grades 7-12 were not slated to be developed until oubse- 
quertt years, assessment in the secondary schools provided baseline data for future 
evaluation efforts as well as input into the needs assessment for the projected curric- 
ulum development. 

No systematic differences between responses in treatment and control schools were 
observed in the initial testing. After less than a year's experience with the program 
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(an instructional plan designed to effect change in knowledge, attitudes, and practices 
over the 13-year school experience), students in treatment schools reflected superiority 
over their peers in control schools. Students in treatment schools were superior in 
knowledge at Grades K, 1, 4, and 6 (a majority of the grades tested). They also were 
superior in attitudes at Grades 1, 3, 4, and 6 (again a majority of the grades tested). 
Superiority of treatment school students was reflected in practices for Grades 4 and 
6. Thus, some differences favoring the students in the treatment group were found for 
students in five of the seven relevant g^ade levels. 

In several respects, this evaluation was a conservat~ e one. For example, the 
evaluation was based on the field test copy of the instructional plan, a document that 
subsequently has been revised and refined. The amount of training received by school 
personnel (one-day workshops) was an additional limitation. Furthermore, because this 
was the first year for use of the plan, students had been exposed to less than one 
- school year of the plan; this limitation is particularly critical because the model 
for the plan is a sequential and integrated 13-year program. The results of the evalu- 
ation are particularly striking in consideration of these limitations and may be seen 
as evidence of the strength and effectiveness of both the treatment and the evaluation 
plan. 

Evaluation Constraints 
From this project description, one might conclude that if only program evaluators 
are given "a chance to do it right," the task is a relatively simple one. A closer 
look at the situation, however, reflects the fallacy of this generalization- 

In fact, the chance to do it right involved far more than being in the right place 
at the right time. The advantages in the situation often came disguised as problems 
to be solved, and there were tradeoffs inherent in the choices associated with each 
opportunity. The result was a situation in which the evaluators actively created "a 
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chance to do it right." Although the specifics were unique to this situation, the 
general pattern is one that may be used by other educational evaluators to create "a 
chance to do it right," 

Establishing a good professional relationship with the state program coordinator 
was a key to gaining the flexibility to develop an adequate eval uation design. The 
evaluators assisted the state program coordinator with the task of devising measurable 
objectives for Tennessee's State Plan for Nutrition Educat.on. This process facili- 
tated the development of confidence in the relationship and helped establish a founda- 
tion for improvement of future evaluation endeavors. Collection of baseline data, 
switching from only a pretest-posttest design to a control group design, and develop- 
ment of instruments compatible with program objectives all were modifications from 
the original plan that were made possible by the relationship that was established 
with the* state program coordinator. 

Another key to program success was the identification of specific program objec- 
tiveSo At the time the evaluation contract for this project was awarded., objectives 
had not been identified. However, through working with the state program coordinator 
and establishing a structural as well as a functional relationship between the evalua- 
tion project and a second project funded specifically for development of student ob- 
jectives and an instructional plan, the evaluators were able to provide input that 
facilitated the evaluation task. The two projects had different project directors 
and different staffs, but there was coordination between the two not jusl through 
communication between the project directors but also through a small core group of 
personnel involved in bpth projects. These persons had technical expertise in program 
content as well as in evaluation methodology. Thus, the advantages of having an inde- 
pendent evaluation team were combined with the advantages of having evaluators with 
first-hand understanding of and opportunity for input into program development. 
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Limitations in project time frames often are problematic, and this nutrition 
education evaluation was no exception* The difficulty was confounded by the neces- 
sity for coordination between the two projects (objectives and evaluation). In fact, 
although the total time available for doing the evaluation was tight, this constraint 
was compounded by the inherent interdependence of the two projects. Differences in 
priorities within the two projects posed potential difficulties, but this disadvan- 
tage was turned into an advantage in some respects. Having a core group of personnel 
involved in both projects facilitated resolution of potential difficulties; the inter- 
action within and between staffs of the two projects and the sharing of resources pro- 
vided a flexibility that would have been impossible to achieve otherwise. Although 
the plan resulted in heavy demands on the individuals involved across the two pro- 
jects, the "time-sharing" concept applied to project personnel facilitated accomplish- 
ment of the overall task. 

Another characteristic of the situation that was both an advantage and a disad- 
vantage was the interdisciplinary nature of the project. The involvement of profes- 
sionals from several disciplinary areas and eight different academic units was impor- 
tant to optimal accomplishment of the task. The practical difficulties of coordinat- 
ing a group with such different perspectives, working styles, and schedules was no 
small consideration. Although there were temptations to limit the diversity repre- 
sented—or at least to divide into relatively homogeneous teams, the effectiveness of 
the process and the product were a function of the extent of the truly interdisciplin- 
ary work involved. 

Perhaps the most pervasive problem in a comprehensive large-scale evaluation is 
that of maintaining effective communication among all of the participants. In this 
project, the management team had to keep communication channels open among faculty in 
several departments in each of two colleges. Public school officials and administra- 
tors, and later teachers and food service personnel /all had to be convinced of the 

er!c Q 



importance of their participation in the evaluation. Field assistants who actually 
traveled to schools throughout the state to conduct the assessment had to be trained 
to convey program purposes accurately to the administrators, teachers, and students 
from whom they were to collect data. The state program coordinator and her staff had 



to be briefed continuously concerning eval uation^tivi ties so they could lend support 



if questions or problems that arose in local schodt^ were brought to their attention. 
The comprehensiveness of the evaluation, which was its greatest strength, created the 
need for an effectively functioning communication network of formidable proportions. 



The resulting evaluation project was not a perfect one at all. In some ways, it 
illustrates a textbook model of program evaluation, but to look beneath the surface 
brings into focus a variety of potentially serious difficulties. Being in the right 
place at the right time is important to good program evaluation; it is a necessary 
condition—but not a sufficient one. Equally important are the willingness and ability 

to identify potential weaknesses and turn them into strengths in the evaluation pro- 

* 

cess. Each project has its 4 own unique constraints, but the sensitivity of educational 
evaluators to the idea that "a chance^jto do it right" is created rather than just 
given is an important step toward good program evaluation. 




Conclusion 



9 



